Week 9 of 12 · The Lead

The Quality-Ops Loop

Synthesize the week into a playbook a new lead could run on Monday — and see where it hands off to calibration

Day 45 ~60 minutes Review

Day 45 of 66

What you now run

You started the week with a label set you thought was "done." You end it owning the operation that keeps it alive. Say the spine back without looking:

The Week 9 spine

The quality-ops playbook

Synthesize the week into something a brand-new lead could pick up and run with no further explanation. This is the deliverable: a one-page operating procedure, not a concept.

The playbook — a new lead could run this Monday
  1. Every week, pull a fresh random sample (~100 items/task) from recent labels. Never let anyone choose which work gets audited.
  2. Score it against gold and run qa_audit.py. Record accuracy, its 95% CI, and inter-annotator agreement.
  3. Compare to the trend. If accuracy drops >5 pts below baseline (or declines three weeks running), it's drift — investigate.
  4. When investigating, check the two cheap suspects first: recent guideline changes, then new annotators. Slice the audit by both.
  5. Adjudicate the disagreements. Named adjudicator resolves each, classifies the cause, and patches the guideline / gold / person.
  6. Send the weekly report. Every number paired with an action; a "what changed and why" narrative on top.
  7. Repeat next week against a fresh sample. The fixes from step 5 are what step 1 re-tests.
Why this is the heartbeat of the job

Every other artifact you've built — the rubric, the pipeline, the preference data — is a thing. This is a rhythm. A Human Data Lead is judged not on whether they once built a clean dataset, but on whether quality is trending up under their watch, week after week. The playbook is the rhythm that makes that true.

Where this hands off — preview of Week 10

Notice that two of the playbook's fixes — "new annotators" and "calibrate that person" — aren't really data problems. They're people problems. The audit can tell you who is drifting and in which direction, but resolving it means a calibration conversation, an onboarding gate, a coaching session.

QA → calibration

Week 9 gave you the instrument that detects a miscalibrated person. Week 10 gives you what to do with them: run calibration sessions that pull a team toward agreement, onboard new annotators so they clear a quality bar before live work, and give feedback that improves labels without crushing morale. The audit hands the calibration session its agenda — they're the same loop, seen from the people side.

Self-quiz (no notes)

Answer out loud or in writing. If any answer is shaky, reread that day before Week 10.

  1. Name the three forces that make data quality drift, and give one concrete example of each.
  2. Your weekly audit reads 88% on 100 items. Roughly how wide is the 95% CI, and why does sample size matter?
  3. Accuracy goes 90% → 87% → 86% over three weeks, each step inside the noise band. Drift or noise? Why?
  4. An alert fires. What are the first two things you check, and why those two?
  5. An adjudicator decides a disputed item is "accept." What must happen next for the resolution to count as done?
Answer key (peek after you try)

1) Guidelines age (new edge case uncovered), annotators fatigue/shortcut (default to "accept" under load), input distribution shifts (new image model changes the outputs). 2) Roughly ±6–7 points; the band is ±1.96·√(p(1−p)/n), and n is under a square root, so bigger samples shrink it (4× n halves the band). 3) Drift — three weeks the same direction is a trend even when each step is inside the band; direction beats step size. 4) Recent guideline changes and new annotators — the cheapest to check and the most common causes; slice the audit by both. 5) The resolution must feed back — update the guideline (a worked example/rule), fix the gold, or schedule that annotator's calibration; otherwise the case recurs.

Close the week

Wrap-up

Before Week 10

Up next — Week 10: Leading Annotator Teams

This is the week the title "Lead" becomes literal. The audit can tell you who's drifting; now you learn what to do about it — calibration sessions, onboarding gates, constructive feedback, and reporting throughput-vs-quality up the chain. You've been the person being calibrated; next week you design the calibration.

The lead move

An IC builds a clean dataset once and points to it as proof of skill. A lead owns a rhythm, not a thing — and is judged on whether quality is trending up week after week under their watch, knowing the audit's hardest findings hand off to the people side, not another data patch.

Say this in an interview: "I'm not measured on whether I once shipped a clean dataset — I'm measured on whether quality is rising under my watch, so I run a one-page quality-ops playbook a new lead could pick up Monday, and I know when a drift signal is really a calibration problem to coach, not a label to redo."

Week 9 Takeaways